{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3D Object Detection Evaluation Tutorial\n",
    "\n",
    "Welcome to the 3D object detection evaluation tutorial! We'll walk through the steps to submit your detections to the competition server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from av2.evaluation.detection.eval import evaluate\n",
    "from av2.evaluation.detection.utils import DetectionCfg\n",
    "from pathlib import Path\n",
    "from av2.utils.io import read_feather, read_all_annotations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Constructing the evaluation configuration\n",
    "\n",
    "The `DetectionCfg` class stores the configuration for the 3D object detection challenge.\n",
    "\n",
    "- During evaluation, we remove _all_ cuboids which are not within the region-of-interest (ROI) which spatially is a 5 meter dilation of the drivable area isocontour. \n",
    "\n",
    "- **NOTE**: If you would like to _locally_ enable this behavior, you **must** pass in the directory to sensor dataset (to build the raster maps from the included vector maps)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_dir = (\n",
    "    Path.home() / \"data\" / \"datasets\" / \"av2\" / \"sensor\"\n",
    ")  # Path to your AV2 sensor dataset directory.\n",
    "competition_cfg = DetectionCfg(\n",
    "    dataset_dir=dataset_dir\n",
    ")  # Defaults to competition parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "split = \"val\"\n",
    "gts = read_all_annotations(\n",
    "    dataset_dir=dataset_dir, split=split\n",
    ")  # Contains all annotations in a particular split.\n",
    "display(gts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing detections for submission.\n",
    "\n",
    "The evaluation expects the following 14 fields within a `pandas.DataFrame`:\n",
    "\n",
    "- `tx_m`: x-component of the object translation in the egovehicle reference frame.\n",
    "- `ty_m`: y-component of the object translation in the egovehicle reference frame.\n",
    "- `tz_m`: z-component of the object translation in the egovehicle reference frame.\n",
    "- `length_m`: Object extent along the x-axis in meters.\n",
    "- `width_m`: Object extent along the y-axis in meters.\n",
    "- `height_m`: Object extent along the z-axis in meters.\n",
    "- `qw`: Real quaternion coefficient.\n",
    "- `qx`: First quaternion coefficient.\n",
    "- `qy`: Second quaternion coefficient.\n",
    "- `qz`: Third quaternion coefficient.\n",
    "- `score`: Object confidence.\n",
    "- `log_id`: Log id associated with the detection.\n",
    "- `timestamp_ns`: Timestamp associated with the detection.\n",
    "- `category`: Object category.\n",
    "\n",
    "Additional details can be found in [SUBMISSION_FORMAT.md](../src/av2/evaluation/detection/SUBMISSION_FORMAT.md)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you've already aggregated your detections into one file.\n",
    "dts_path = Path(\"detections.feather\")\n",
    "dts = read_feather(dts_path)\n",
    "\n",
    "dts, gts, metrics = evaluate(dts, gts, cfg=competition_cfg)  # Evaluate instances."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, if you would like to submit to the evaluation server, you just need to export your detections into a `.feather` file. This can be done by:\n",
    "\n",
    "```python\n",
    "dts.to_feather(\"detections.feather\")\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "b5ba47cd2ceb6b4b8c5d326eb375c1eb469e3d27b6f068302112bb12ba7862a5"
  },
  "kernelspec": {
   "display_name": "Python 3.9.12 ('av2')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
